Development of complex software systems tend to be time consuming and difficult to complete because of frequently changing customer requirements. Traditional software development project management theory often mandates the creation of a “requirements” document that describes the functionality and features of the software system to be developed. A software development organization will create the requirements document based on input from the customer, and then have the customer approve the document so as to “lock” in the expectations of the customer and conduct the development work against a plan that does not change.
Unfortunately, customers often do not have a firm idea of all their requirements until after development has started. Also, changing business needs will often dictate changes in system requirements as the software development project is underway. The result is often frustration for both software developers and their customers. Software developers want some finality in the requirements document, so that customer requested changes do not lead to continually redesigning or redoing development work without ever getting to project completion. Customers, on the other hand, desire (and sometimes feel compelled to make) changes in requirements as their needs change, especially in large projects where at the end of a development period the needs of a business may have change so dramatically from those in the beginning, that the end work product will be outdated and not useful if the requirements document remains static from start to finish.
One recent change in software development processes to address the forgoing problems has been the use of Extreme Programming (XP). A complete description of the XP process can be found in Beck, K., Extreme Programming Explained: Embrace Change (Boston, Mass., Addison-Wesley 2000) and Auer, K. and Miller, R., Extreme Programming Applied: Playing to Win (Boston, Mass., Addison-Wesley 2002).
In an XP process, a complete project requirements document is eliminated in favor of a more dynamic process, in which customers help define smaller more discrete sets of requirements (referred to as “user stories”), which are each a short description (perhaps no more than a few sentences) as to what the customer needs. The software code to implement the user story is intended to be developed within a short interval of time, say, two weeks (which interval is referred to as an “iteration”). During each iteration, there may several user stories to be completed. The number of hours (often referred to as “velocity” in the XP process) estimated for development of user stories in an iteration are matched to the development resources (developer hours or velocity) available for that iteration. For flexibility, the customer may develop new user stories as time goes along to incorporate changing requirements, and there is continuous collaboration and feedback between the developer and the customer. As a result, not only are requirements met shortly after articulated by the customer, but the customer also gets more realistic estimates of completion dates for each “user story” in the project as the user stories are created.
While the XP process is often appropriate for smaller development projects, it is less useful for a project that is complex, involving a large, costly set of requirements with a large number of individual developers. This is especially true when there are legacy or existing systems with which the developed software system will need to work.
One problem with XP in a larger development project is that corporate governance considerations often require that the process be formalized in a written document (or other tangible form) that can be used to support the scope and cost of the project should it be questioned by shareholders or regulators. A further problem with XP in a large project is the common need for the developed software system to work with existing systems. In such cases, the amount of testing of the developed system increases dramatically (all components, i.e., user stories, of the developed system need to be tested not only for individual acceptance, but also tested to make sure they work with each other, and further to make sure they work with every one of the existing systems). The XP process, because of its emphasis on efficiency and less formality, tends to look at testing as an extension of development, to be articulated and worked on in the same way and at the same time as actual development. Complex testing requirements (involving interrelationships with existing systems) are frequently overlooked or underestimated.